Method and system for estimating information related to a vehicle pitch and/or roll angle

ABSTRACT

The present disclosure relates to a method ( 200 ) for estimating information related to a vehicle pitch and/or roll angle. The method comprises a step of obtaining ( 220 ) a first estimate of the information related to the pitch and/or roll angle. The method is characterized by the steps of capturing ( 210 ) an image of an area covering at least a part of the horizon using a camera mounted on the airborne vehicle, and determining ( 240 ) an improved estimate of the information related to the pitch and/or roll angle based on the first estimate of the information related to the pitch and/or roll angle, and a digital elevation model.

TECHNICAL FIELD

The present disclosure relates to a method for estimating informationrelated to a vehicle pitch and/or roll angle. The present disclosureparticularly relates to a method for estimating information related to avehicle pitch and/or roll angle, wherein a first estimate of theinformation related to the pitch and/or roll angle is obtained.

The present disclosure further relates to a system for estimatinginformation related to a vehicle pitch and/or roll angle.

BACKGROUND

For navigation of unmanned aerial vehicles (UAVs) continuous absoluteestimation/determination of the vehicle orientation (attitude) iscrucial. Today, the most common solution to obtain this information issensor fusion of data from GPS and inertial sensors (IMUs).

There are several image based methods proposed in the literature wherehorizon detection is used for vehicle attitude estimation.

Gyer M. Methods for Computing Photogrammetric Refraction for Verticaland Oblique Photographs. Photogrammetric Engineering and Remote Sensing,1996 describes refraction of light in the atmosphere.

SUMMARY

The attitude accuracy obtainable with a system comprising of a GPS and ahighly accurate IMU is sufficient in most airborne navigation problems.However, GPS reception may not always be available in certainenvironments and in a conflict situation.

Also, highly accurate IMUs often weigh a few kilos and may exceed thepayload limit for lighter airborne vehicles. The accurate IMUs are alsovery costly and are often subject to export restrictions.

All IMU's suffer from drift, i.e. errors accumulate over time, resultingin unreliable absolute measurements with time. Lighter and less costlyIMU's have larger drift and need support from additional sensors tomaintain an accurate absolute orientation estimate over time.

One objective of the present disclosure is therefore to improve attitudeaccuracy. This has in one embodiment been achieved by estimatinginformation related to absolute vehicle pitch and roll angles from imagehorizon detection and digital elevation model data.

One embodiment of the present disclosure relates to a method forestimating information related to a vehicle pitch and/or roll angle. Themethod comprises a step of obtaining a first estimate of the informationrelated to the pitch and/or roll angle. The method further comprises astep of capturing an image of an area covering at least a part of thehorizon using a camera mounted on the airborne vehicle, and a step ofdetermining an improved estimate of the information related to the pitchand/or roll angle based on the first estimate of the information relatedto the pitch and/or roll angle, and a digital elevation model.

This method provides improved real-time capability that uses a digitalelevation model (DEM) to improve the accuracy of the estimate of pitchand/or roll angle related information.

The proposed invention can be used as a stand-alone sensor for absolutevehicle attitude or three dimensional position estimation or as part ofa sensor fusion network.

The accuracy of the method will depend on the resolution of the cameraand the DEM used in the implementation. That accuracy will be obtainablethroughout the complete flight and will not degrade with flight time incontrast to an inertial device.

The concept of the method allows the camera used to be either an opticalsensor or an IR sensor. Thereby, the system can be used at day or night.

In one option, the camera is an omni-directional camera.

In one option, the first estimate of the information related to thepitch and/or roll angle is obtained based on a first horizon estimate,wherein the earth is assumed to be spherical with no topography.

In one option, the method further comprises a step of obtaining at leastone second horizon estimate based on the digital elevation model. Thedetermination of the improved estimate of the information related to thepitch and/or roll angle estimate is then based on the obtained at leastone second horizon estimate and preferrably the first horizon estimate.

In one option, the step of determining the at least one second horizonestimate comprises the following steps:

-   -   obtaining the respective elevation profile for a plurality of        angular directions α_(i) around the vehicle from the digital        elevation model based on a vehicle and/or camera three        dimensional position,    -   determining the largest incidence angle on the camera, that is        generated by all land objects at heights and distances given by        the extracted elevation profile, and    -   determining for each angular direction α_(i) along the        respective elevation profile the maximum of all incidence angles        to be the geometrical effective horizon point.

In one option, a plurality of candidate second horizon estimates aredetermined each related to a candidate camera and/or vehicle threedimensional position. In accordance with this option, the determinationof an improved estimate of the information related to the pitch and/orroll angle involves selecting that candidate second horizon estimatewhich provides a best fit between the first horizon estimate and thesecond horizon estimate.

In one option, the improved pitch and/or roll angle related estimateobtaining element is arranged to obtain the camera and/or vehicle threedimensional position related to the the selected candidate secondhorizon estimate as the true camera and/or vehicle three dimensionalposition.

In one option, the camera and/or vehicle three dimensional positionrelated to the the selected candidate second horizon estimate isdetermined to be the true camera and/or vehicle three dimensionalposition.

In one option, the determination of the largest incidence angle on thecamera, which is generated by all land objects at heights and distancesgiven by the extracted elevation profile, is performed based on anextensive search. In an alternative example, the determination isperformed based on a look-up table comprising information related toincidence angles on the camera for substantially all objects along theelevation profile. The look-up table comprises in one option inputparameters related to camera and/or vehicle three dimensional position,object height and/or distance from the camera; and an output parameterrelated to incidence angle on the camera. In one example the position ofthe camera and/or vehicle is known in an XY plane and the inputparameter is height of the camera and/or vehicle.

In one option, the step of obtaining the first estimate of the pitchand/or roll angle related information comprises obtaining the firsthorizon estimate based on the captured image. In this option, the stepof determining an improved estimate of the pitch and/or roll anglerelated information comprises adjusting the first estimate of the pitchand/or roll angle related information based on a relation between thefirst horizon estimate and the second horizon estimate.

In one option, the determination of the first estimate of the pitchand/or roll angle related information comprises back projectingdetermined horizon edge pixels in the image onto the unit sphere, anddetermining the first estimate of the pitch and/or roll relatedinformation based on the back projected edge pixels.

In one option, the determination of the first estimate of the pitchand/or roll related information based on the back projected edge pixelscomprises probabilistic voting such as probabilistic Hough voting forall edge pixels.

In one option, the first horizon estimate is determined based on theback projected edge pixels.

In one option, the method further comprises steps of extracting thoseedge pixels which are within a determined range from the first horizonestimate and projecting the extracted edge pixels onto the unit sphereso as to provide an updated first horizon estimate. In this option, thestep of determining an improved estimate of the pitch and/or rollrelated information comprises adjusting the first estimate of the pitchand/or roll related information based on a relation between the updatedfirst horizon estimate and the second horizon estimate.

The present disclosure also relates to a system for estimatinginformation related to a vehicle pitch and/or roll angle. The systemcomprises a processing unit having a first pitch and/or roll anglerelated estimate obtaining element arranged to obtain a first estimateof the information related to the pitch and/or roll angle. The systemfurther comprises at least one camera mounted on the airborne vehicle.The camera is arranged to capture an image of an area covering at leasta part of the horizon. The system further comprises a database arrangedto store a digital elevation model. The processing unit comprisesfurther an improved estimate obtaining element arranged to determine animproved estimate of the information related to the pitch and/or rollangle based on the first estimate of the pitch and/or roll angle relatedinformation, and the digital elevation model.

In one option, the first pitch and/or roll angle related estimateobtaining element is arranged to obtain the first estimate of the pitchand/or roll angle related information based on a first horizon estimate,wherein the earth is assumed to be spherical with no topography. In oneexample, the first horizon estimate is determined based on the image.

In one option, the improved pitch and/or roll angle related estimateobtaining element is arranged to obtain at least one second horizonestimate based on the digital elevation model, and to determine theimproved estimate of the pitch and/or roll angle related informationbased on the obtained at least one second horizon estimate.

In one option, the improved pitch and/or roll related estimate obtainingelement is arranged to determine the second at least one horizonestimate by extracting the elevation profile in all angular directionsα_(i) around the vehicle from the digital elevation model based on avehicle three dimensional position, determining the largest incidenceangle on the camera, that is generated by all land objects at heightsand distances given by the extracted elevation profile, and determiningfor each angular direction α_(i) the maximum of all incidence anglesalong the elevation profile to be the geometrical horizon point wherethe elevation profile is valid.

In one example, the system comprises means for performing an extensivesearch so as to determine the largest incidence angle on the camera.Alternatively, the system comprises a look-up table comprisinginformation related to incidence angles on the camera for substantiallyall objects along the elevation profile. The improved estimate obtainingelement is then arranged to determine the largest incidence angle on thecamera, which is generated by all land objects at heights and distancesgiven by the extracted elevation profile based on the look-up table.

In one option, the first pitch and/or roll angle related estimateobtaining element is arranged to obtain a first horizon estimate basedon the captured image. The improved pitch and/or roll angle relatedestimate obtaining element is then arranged to adjust the first estimateof the pitch and/or roll angle related information based on a relationbetween the first horizon estimate and the at least one second horizonestimate so as to determine the improved estimate of the pitch and/orroll angle related information.

The first pitch and/or roll angle related estimate obtaining element isin one option arranged to back project determined, detected edge pixelsonto the unit sphere, and to determine the first estimate of the pitchand/or roll angle related information based on the back projected edgepixels.

The first pitch and/or roll angle related estimate obtaining element isin one option arranged to perform probabilistic voting such asprobabilistic Hough voting for all edge pixels so as to determine thefirst estimate of the pitch and/or roll angle related information basedon the back projected edge pixels.

The first pitch and/or roll angle related estimate obtaining element isin one option further arranged to determine the first horizon estimatebased on the back projected edge pixels.

In one option, the first pitch and/or roll angle related estimateobtaining element is arranged to extract those edge pixels which arewithin a determined range from the first horizon estimate and to projectthe extracted edge pixels onto the unit sphere so as to provide anupdated first horizon estimate. The improved pitch and/or roll anglerelated estimate obtaining element is then arranged to adjust the firstestimate of the pitch and/or roll angle related information based on arelation between the updated first horizon estimate and the secondhorizon estimate so as to determine the improved estimate of the pitchand/or roll angle related information.

One embodiment of the present disclosure relates to a computer programfor executing the steps of the method for estimating information relatedto a vehicle pitch and/or roll angle according to the above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates schematically an example of an airborne vehicle.

FIG. 2 is a flow chart illustrating an example of a method forestimating vehicle pitch and/or roll angle related information in anairborne vehicle.

FIG. 3 is a flow chart illustrating an example of a step for determininga second horizon estimate based on a digital elevation model.

FIG. 4 illustrates schematically a principle for forming a look-up tablefor obtaining information related to an incidence angle on a camera.

FIG. 5 is a flow chart illustrating an example of a step for obtaining afirst estimate of information related to pitch and/or roll angle.

FIG. 6 is a flow chart overview illustrating an example a flow ofestimating vehicle pitch and/or roll angle related information in anairborne vehicle.

FIG. 7 illustrates a principle for determining a geometrically computedhorizon.

FIG. 8 illustrates an example of estimating a horizon normal.

FIG. 9 illustrates an example of a band on the unit sphere withconceivable true horizon edge pixels.

FIG. 10 is a block scheme schematically illustrating an example of asystem for estimating vehicle pitch and/or roll angle relatedinformation in an airborne vehicle.

DETAILED DESCRIPTION

In FIG. 1, an airborne vehicle 100 comprises a camera 101. The camera isarranged to capture at least one image. In one example, the camera isarranged to continuously capture images of a scene. A processing unit102 is in one example arranged to determine a vehicle pitch and/or rollangle related information. In one example the vehicle pitch and/or rollangle related information comprises a vehicle pitch and/or a vehicleroll value. In one example the vehicle pitch and/or roll angle relatedinformation comprises information transformable to a vehicle pitchand/or a vehicle roll value. The processing unit 102 is arranged tooperate in cooperation with at least one memory element 103. The atleast one memory element 103 comprises a software algorithm arranged tobe executed by the processing unit 102. The at least one memory element103 is also arranged to store data used by the processing unit 102. Inone example (not shown) at least part of the processing unit 102 and/orthe at least one memory element 103 is located outside the airbornevehicle. In one example, the camera is mounted on the outside of theaircraft.

In one example, the camera 101 is mounted with its optical axis 104vertically down. However, the camera 101 can be mounted with its opticalaxis in other directions.

In one example, the camera is an omni-directional camera. Theomni-directional camera should have a field of view, FOV, such that itis ensured that at least part of the horizon can be seen irrespective ofthe airborne vehicle tilt angle assuming that it is at sufficientheight, i.e. above trees and buildings. In one example, theomni-directional camera has a field of view, FOV, larger than 180degrees. The omni-directional camera has in one example at least oneomni-directional lens. In one example, the at least one omni-directionallens is a fish-eye lens.

In one example, the camera is a camera for visual light. In one example,the camera is an IR camera. In one example, a camera set is providedcomprising one camera for visual light and one IR camera. Then, thatcamera can be used which captures the best quality image given thecircumstances. For example, during night, an IR camera would work betterthan a visual image camera. During a sunny day, the visual camera wouldnormally provide the best image.

In one example, the camera(s) is calibrated, i.e. its intrinsicparameters and the distortion parameters are known. The orientation(rotation) of the camera in a vehicle fixed coordinate frame is known.These prerequisites are for example determined pre-flight in labenvironment or in an earlier flight.

The airborne vehicle 100 stores an elevation database 105 or has accessto the elevation database. In one example, the elevation database 105comprises a digital elevation model, DEM, of an area around the vehicle.

The digital elevation model could be any kind of 3D model known to thoseskilled in the art. In one example, the digital elevation model isrepresented as a mesh. In one example the digital elevation model isrepresented as a surface representation. In one example the digitalelevation model is represented as a voxel representation. In one examplethe digital elevation model includes texture information. In one examplethe digital elevation model is a triangular irregular network(tin)-based mesh draped with textures.

The term digital elevation model is used herein. This term is intendedto include a digital surface model as well as a digital elevation modelwithout any structures modelled.

The airborne vehicle 100 optionally further comprises a heightindicating element 106. The height indicating element 106 is arranged toprovide an approximate height for the vehicle at the time of capturingthe image. In one example, the height indicating element comprises anair pressure meter. In one example, the height indicating element 106comprises a GPS receiver or the like providing the three dimensionalposition of the airborne vehicle in a global coordinate system.

The airborne vehicle optionally further comprises a position and/orheading indicating element 107. The approximate position and heading ofthe vehicle in global coordinate system, or relative to the elevationdatabase 105, is known. The approximate position and/or heading of theairborne vehicle in the global coordinate system is in one exampleobtained from GPS and/or object tracking in an image sequence capturedonboard the airborne vehicle. As stated above, the processing unit isarranged to estimate vehicle pitch and/or roll angle relatedinformation. Processing involves determining a first estimate of thepitch and/or roll angle related information and determining an improvedpitch and/or roll related estimate based on the first estimate, theimage(s) captured by the camera and the digital elevation model.

In FIG. 2, a method 200 for estimating information related to a vehiclepitch and/or roll angle is illustrated. The method comprises a step ofcapturing 210 an image of an area covering at least a part of thehorizon using a camera mounted on the airborne vehicle, obtaining 220 afirst estimate of the pitch and/or roll angle related information, anddetermining 240 an improved estimate of the pitch and/or roll relatedinformation based on the first estimate, the image and a digitalelevation model. In one example, the step of capturing 220 an image isperformed in parallel or after the step of obtaining 210 a firstestimate of the pitch and/or roll angle related information.

In one example, the first estimate of the pitch and/or roll anglerelated information is obtained 220 based on a first horizon estimate,wherein the earth is assumed to be spherical with no topography.

In one example, the step of obtaining 220 the first estimate comprisesobtaining a first horizon estimate based on the captured image.

In one example, the method comprises a step of determining 230 a secondhorizon estimate based on the digital elevation model. The step ofdetermining 240 the improved pitch and/or roll estimate is then based onthe determined second horizon estimate. The step of determining 240 animproved estimate of the pitch and/or roll related information comprisesin one example adjusting the first estimate of the pitch and/or rollrelated information based on a relation between the first horizonestimate and the second horizon estimate.

In FIG. 3, one example of performing a step of determining 330 a secondhorizon estimate based on the digital elevation model is described. Thedetermination of the second horizon estimate comprises in this examplethe following. In a first step, an elevation profile for the airbornevehicle is obtained 331 in a plurality of directions around the airbornevehicle. In one example this involves obtaining a global threedimensional position for the vehicle/camera. The three dimensionalposition is in one example obtained from height indicating elementand/or a position and/or heading indicating element onboard the vehicleor using an image-based method. If GPS position information is notavailable, the accuracy of the three dimensional position is in oneexample known with a lower accuracy in at least one dimension. In oneexample, the accuracy in a XY-plane is in the range of tens of meters.If the max height in a flight is 600 m, the distance to the idealhorizon is around 100 km. The ideal horizon is defined as a horizondetermined based on the assumption that the earth is spherical with notopography. To have some margin, height profiles up to 200 km away fromthe vehicle are in one example obtained. Based on the vehicle/camerathree dimensional position, the height profile in all angular directionsα_(i) around the vehicle is extracted from the DEM. For each direction,the elevation profile is obtained at a predetermined interval. In oneexample, the predetermined interval is within the same range as theresolution in the digital elevation model, DEM, used. In one example,bilinear interpolation is used to find the elevation between grid pointsin the digital elevation model.

In a next step, the largest incidence angle on the camera is determined332, which is generated by all land objects at heights and distancesgiven by the obtained elevation profile. As it would be computationallydemanding to compute ray paths between all objects along the elevationprofile and the camera online, in one example, look-up tables are usedto find the incidence angle on the camera for all land objects along theelevation profile. One example of generation of look-up tables will bedescribed later. In an alternative example, an extensive search is usedto find the incidence angle on the camera for all land objects along theelevation profile. There are a plurality of methods known in the art forfinding the incidence angle on the camera for all land objects along theelevation profile. Below, the search is described in relation to alook-up table. In a next step, a maximum of all incidence angles alongthe elevation profile, denoted ξ_(max) are determined 333 to be thegeometrical horizon point for the angular direction α_(i) where theelevation profile is valid. This is repeated for all angular directionsα_(i) in the XY plane. Thereby, a geometrically computed horizoncomprises i points each given as

P _(geom,i)=[cos α_(i) sin α_(i) cos(π−ξ_(max,i))]^(T)

Different examples related to generation of look-up tables are describedbelow. In one example, the determining 332 of the largest incidenceangle on the camera, that is generated by all land objects at heightsand distances given by the obtained elevation profile is performed basedon a look-up table comprising information related to incidence angles onthe camera for substantially all objects along the elevation profile. Inone example the input parameters comprise camera three dimensionalposition and/or object height and/or distance from the camera. In oneexample, the output parameter is the incidence angle on the camera. Thisprocedure may be performed for a plurality of sets of input parameters.Especially, the procedure may be performed for a plurality of possiblethree dimensional positions.

Increments are selected for each input parameter. In one example theincrement for the camera height parameter is 2 meters. In one examplethe increment for the object height is 2 meters. In one example theincrement for the distance from the camera is 100 meters.

The digital elevation model, DEM, has a predetermined XY gridresolution. In one example, the increments for the input parameters arein the same order as the digital elevation model resolution. In oneexample, the resolution is 3 arc-seconds or around 90 m at the equator.

In FIG. 7, a principle for determining a second, geometrically computedhorizon estimate comprising i points P_(i) as given in the equationP_(geom, i)=[cos α_(i) sin α_(i) cos(π−ξ_(max, i)]^(T) is illustrated.In the figure, the camera three dimensional position is represented asthe origin of the coordinate system. The z-axis represents height. Thex- and y-axes are defining a ground plane. In the illustrated example,α_(i) represents an angular direction to point P_(i) in the XY plane. Inthe illustrated example, the XY plane angular direction α_(i) is givenin relation to the x-axis. In the illustrated example, α_(i) representsan angular direction to point P_(i) in the XY plane. ξ_(max, i)represents the largest incidence angle on the camera that is generatedby all land objects at heights and distances given by an extractedelevation profile. In the illustrated example, the largest incidenceangle ξ_(max, i) represents an angle between the z-axis and a linebetween the origin and point P_(i).

In FIG. 4, a principle for obtaining incidence angles to use for examplein a look-up table comprising at least one of the input parameters:camera three dimensional position or height, object height, and distancefrom the camera, is illustrated by means of an example.

In FIG. 4 a, a refracted ray path is represented with ideal horizon atsea level h₀. FIG. 4 b represents a refracted ray above ideal horizonpassing an object at height h_(obj) located the distance d_(obj) fromthe camera. FIG. 4 c represents a refracted ray above ideal horizonpassing an object at height h_(obj) located beyond the ideal horizon.

In detail, for a spherically stratified model, an angle θ_(c) from aground point at height h_(g) to a camera at height h_(c) is given by

$\theta_{C} = {\int_{h_{g}}^{h_{c}}{\frac{k}{r\sqrt{{n^{2}r^{2}} - k^{2}}}{{r}.}}}$

The subscripts c and g denote the camera and the ground, respectively,and n denotes the refractive index.

The stratified model comprises a model of the refractive index n as afunction of the height h. The model comprises thin radial layers ofthickness Δr with constant refractive index n. The stratified modeldetermines the refracted ray path from the ideal horizon at sea level upto the camera at height h_(c).

For a thin layer i with constant refractive index n_(i) in thespherically stratified model, the integral above may be expressed as

${\theta_{C_{i}} = {{\int_{h_{i}}^{h_{i + 1}}{\frac{k/n_{i}}{r\sqrt{r^{2} - \left( {k/n_{i}} \right)^{2}}}{r}}} = {{\cos^{- 1}\left( \frac{k}{{n_{i}r_{i}} + 1} \right)} - {\cos^{- 1}\left( \frac{k}{n_{i}r_{i}} \right)}}}},$

The camera height h_(c) is set constant, for example within the range200 m to 600 m.

In FIG. 4 a, it is assumed that the earth is spherical with notopography. The perceived horizon will then be at sea level, i.e. atheight h₀=0. At the horizon, the incidence angle is ξ₀=π/2. The equationabove is used to compute angular increments θ_(ci) for each layer ofthickness dr up to the height h_(c). Adding these incremental steps willyield the distance to the ideal horizon

${d_{id}\left( h_{c} \right)} = {\sum\limits_{h_{i} = 0}^{h_{c}}{\theta_{C_{i}}{r_{i}.}}}$

The corresponding incidence angle on the camera ξ_(id)(h_(c)) arecomputed from a model as a sphere with a radius R_(e) and a groundtopography overlaid on this sphere. For refracted ray propagation in theatmosphere around the earth, the above described spherically stratifiedmodel is used. The model determines the refracted ray path from theideal horizon at sea level up to the camera at height h_(c). If therewould be no topography on a spherical earth with radius R_(e), this raypath would give the perceived horizon.

The true perceived horizon (sky/ground boundary) may be shifted in theimage compared to the ideal horizon for two reasons. The first reason isthat there is a land object at higher height than the ray path withinthe distance to the ideal horizon. The second reason is that there is aland object further away than the distance to the ideal horizon, but atsufficient height to be perceived at an incidence angle above the idealhorizon.

In FIG. 4 b, a first case illustrates the effect of high objects withindistance to ideal horizon. To compute this effect, a ground height h_(g)between 0 and h_(c) is considered. The incidence angle at the groundobject is set to ξ_(g)=π/2. The ray path is computed in the same manneras above until it reaches the height h_(c). Data points from the raypath are extracted at desired land object heights h_(obj) and thecorresponding distances to the camera d_(c, obj) and the ray incidenceangle on the camera The computations for all ground heights h_(g) arerepeated from 0 to h_(c) in steps Δh_(g).

In FIG. 4 c, a second case illustrates the effect of high objects beyonddistance to ideal horizon. Even if an object is further away from thecamera than the distance to the ideal horizon, the object may shift thesky/ground limit from the ideal case. The ray from the camera may take apath as in FIG. 4 c, where the ray starts at point P_(obj), lowers inheight from the ground object until it reaches the incidence angle π/2at point P₉₀ and then increases in height again until it reaches thecamera at point P_(c). To compute this effect, we start with a ray atheight h₉₀ and set the incidence angle ξ₉₀=π/2. We compute the ray pathup to the maximum of the camera height h_(c) and the highest objectheight h_(obj, max) in the digital elevation model, DEM. From this raypath, the distance to the camera d₁ from P₉₀ and the corresponding rayincidence angle on the camera ξ_(c,obj) is extracted. From the ray pathon the right side of P₉₀, the distances d₂ to the desired object heightsh_(obj) are extracted. The distances d₁ and d₂ are summed to obtain thetotal distance d_(obj) from the camera to the object at height h_(obj).The total distance d_(obj) is recorded together with the correspondingincidence angle ξ_(obj). The computations are repeated for the heighth₉₀ from 0 up to h_(c).

From the first and second cases of FIGS. 4 b and 4 c, a point set withincidence angles and distances from the camera to the object can beextracted for each camera height h_(c) and object height h_(obj). Thispoint set is resampled to obtain incidence angles at the desired stepsize for the distance. In one example the step size is every 100 m up to200 km. In one example, the look-up table is then generated. Inputparameters to the exemplified look-up table are then camera threedimensional position, object height, distance from camera and the outputparameter is the incidence angle on the camera.

In this example, objects that are at a higher height than the camera andwithin a distance that is shorter than the distance to the ideal horizonhave not been taken into account. If flying in valleys in a mountainousarea, it would be necessary to take this into account.

In FIG. 5, a step of obtaining 520 a first estimate of informationrelated to pitch and/or roll angle.

The method comprises a step of back projecting 522 determined edgepixels in the image onto the unit sphere. Thereafter, the first estimateof the pitch and/or roll angle related angle is determined 523 based onthe back projected edge pixels. Further, in one example, an updatedfirst horizon estimate is determined 524 based on the back projectededge pixels.

The method comprises in one example determining 521 edge pixels in theimage that are within a predetermined distance from a horizon line inthe image.

The step of determining 521 edge pixels comprises in one exampleapplying an edge detector in the input image. Several edge detectors areavailable in the literature. For example, the Canny detector is asuitable choice. An edge direction for the respective determined edgepixel is in one example also determined.

The step of back projecting 522 the determined detected edge pixels ontothe unit sphere comprises, in one example projecting each edge pixelonto the unit sphere. In one example, an estimated pitch and/or rollangle proposed by the respective edge pixel is computed given an assumedheight, measured edge direction and assuming a spherical earth with notopography.

The step of determining 523 the first estimate of the pitch and/or rollrelated information based on the back projected edge pixels comprises inone example performing probabilistic voting such as probabilistic Houghvoting for all or a subset of the edge pixels. The votes from the edgepixels in a Hough voting scheme are accumulated. A cell with the highestvoting score gives the first estimate of the pitch and/or roll anglerelated information. Further, a first horizon estimate is determinedbased on the first estimate of the pitch and/or roll angle relatedinformation. Determining the first horizon estimate involves projectinghorizon pixels on the image plane using the camera calibration. Thus thedetermining of the first horizon estimate forms an inverse of the backprojection made in the back projecting step 522.

The step of determining 524 an updated first horizon estimate based onthe back projected edge pixels comprises in one example extracting thoseedge pixel which are within a determined range around the first horizonestimate and projecting the extracted edge pixels onto the unit sphereso as to obtain the updated first horizon estimate. The range is in oneexample defined by a number of pixels in the image. The determined rangeis for example pixels within a range of maximum of 10 pixels away fromthe first horizon estimate. In an alternative example, the determinedrange is 1 pixel away. The range is in one example defined as an angularband on the unit sphere (FIG. 10). In one example the determined rangeis 0.5°.

In detail, given the first estimate of the pitch and/or roll anglerelated information, all detected edge pixels that are within thedetermined range, for example 1 pixel from the estimated horizon line onthe image are used. Thereafter, all edge pixels within the determinedrange from the estimated horizon line on the image are back projectedonto the unit sphere given the calibrated camera parameters. The pointson the unit sphere are transformed in accordance with the estimatedpitch and roll angles and the heading angle. Thereby, the updated firsthorizon estimate is obtained.

Obtaining of an improved estimate of pitch and/or roll relatedinformation comprises then in one example adjusting the first estimateof the pitch and/or roll related information based on a relation betweenthe first horizon estimate or the updated first horizon estimate, and asecond horizon estimate.

In detail, in one example, the maximum perceived incidence angle on thecamera (unit sphere) for all azimuth angles in the XY plane is thendetermined based on the approximate three dimensional position, headingand height height of the vehicle/camera, the digital elevation model andpossibly the look-up table, so as to obtain the second horizon estimate.Thereafter, the first estimate of the pitch and/or roll relatedinformation is refined/adjusted by minimizing the distance between theback projected and transformed image points and the geometricallydetermined points. The estimated vehicle pitch and roll angle is theresult after the refinement.

FIG. 6 illustrates an example of a flowchart overview. The flow chartcomprises six steps. Steps 1-3 relate to obtaining a first estimate ofthe information related to a pitch and/or roll angle while steps 4-6relate to refinement steps for obtaining an improved estimate of theinformation related to a pitch and/or roll angle.

1. Run an edge detector on the input image. A Canny detector is in oneexample chosen. The Canny detector is robust and known to give nosystematic offset in edge location.2. Estimation of the horizon normal vector on the unit sphere for eachedge pixel. Project the edge pixel onto the unit sphere. The edge pixeldirection on the image plane is projected as a tangent vector on theunit sphere. For a known vehicle height, the radius for the horizon onthe unit sphere is deterministic. Combining this information, thehorizon normal vector, i.e. the vehicle attitude, can be estimated foreach edge pixel.3. Probabilistic Hough voting (Hough, 1962) for all edge pixels toestimate the vehicle attitude (pitch and roll angles) and the horizon inthe image. The voting allows the use of probability density functionsfor the vehicle height, and pitch and roll angles to make the votingmore robust.4. Extract edge pixels close to the estimated horizon on the image planeand project the extracted horizon edge pixels onto the unit sphere.5. Compute the geometrical horizon on the unit sphere from a digitalelevation model using an approximate vehicle position and heading asinput.6. Refine the estimated vehicle attitude from step 3 by minimizing thedistance between the perceived horizon pixels from step 4 with thegeometrically computed horizon from step 5.

The first, edge detecting step comprises in one example at least some ofthe following features. Before applying the Canny detector, the colourimage is in one example converted to greyscale. The image is in oneexample smoothed. In one example the smoothing is performed using aGaussian kernel. To reduce subsequent unnecessary computations in theHough voting, in one example, edge pixels originating from the fisheyecircle and/or aircraft structure are removed.

The second step of estimating a horizon normal vector is illustrated inFIG. 7. The second step comprises in one example at least some of thefollowing. For an image edge pixel p=(x,y), the projection onto the unitsphere is at point P. The gradient is computed in p, and the edgedirection is defined in p as (−∇_(y), ∇_(x)), i.e. normal to thegradient. The image point p_(e) is defined as the point one pixel awayfrom p along the edge direction. The projection of p_(e) onto the unitsphere is at P_(e). If p is a horizon point, the vector {right arrowover (PP_(e))} vector is a tangent vector on the unit sphere lying inthe plane of the projected horizon. Let t be a vector of unit length inthe direction of {right arrow over (PP_(e))}. In a cross section of theunit sphere, orthogonal to the vector t, we search for a second point Qin the plane of the horizon. For a certain height h_(c), the radius ofthe horizon circle on the unit sphere is known. To find Q, we define thevector

{right arrow over (OS)}={right arrow over (OP)}×t

where O is the origin in the unit sphere. We then obtain the vector

{right arrow over (OQ)}={right arrow over (OP)}cos 2ξ_(c) +{right arrowover (OS)}sin 2ξ_(c)

where ξ_(c) is the incidence angle from the horizon for a camera atheight h_(c). At this stage, we assume that the earth is smooth withradius R_(e). The points Q_(max) and Q_(min) denote the horizon pointsfor the maximum and minimum heights given the probability distributionp_(h) for the camera height in the subsequent voting.

A unit normal vector {circumflex over (n)} to the horizon plane can nowbe obtained as

$\hat{n} = {\frac{\overset{\rightarrow}{PQ} \times t}{{{{\overset{\rightarrow}{PQ} \times t}}}_{2}}.}$

The pitch and roll angle estimates for the edge point p are then givenby

${\theta = {{arc}\; \sin {\hat{n}}_{y}}},{\varphi = {{- \arctan}{\frac{{\hat{n}}_{x}}{{\hat{n}}_{z}}.}}}$

The third step of performing Probabilistic Hough voting comprises in oneexample at least some of the following. For each edge pixel p, we haveshown how to compute the estimated pitch and roll angles for the horizonplane, given an assumed height h. The accumulator cells in the Houghvoting is chosen as a pitch and roll angle grid. In the probabilisticHough voting scheme, the weight w for each vote is proportional to thelikelihood that the edge pixel is a horizon pixel given the probabilitydistributions p_(h), p_(θ) and p_(φ) for h_(c), θ and φ. Thus,

w(x,y)∝∫∫∫p(x,y|h,θ,φ)dφdθdh

The height can be measured either with a GPS or using an air pressuremeter onboard the vehicle.

Using Bayes' theorem and assuming that the probability distributions forh_(c), θ and φ are independent, the weights may be calculated as

w(x,y)∝∫p _(h)(h)dh∫p _(θ)(θ)dθ∫p _(φ)(φ)dφ=w _(h) w _(θ) w _(φ)

For each edge pixel p, the estimated pitch and roll angles for eachheight h is computed. A weighted vote is in one example given in thenearest neighbour pitch-roll cell in the accumulator array.

In order to suppress local maxima, the values in the accumulator arrayare in one example convolved with a Gaussian kernel. The index for thecell with maximum score is then picked as the attitude estimate.

The fourth step of extracting horizon edge pixels comprises in oneexample at least some of the following.

After step three, there may be present significant attitude estimateerrors due to nonflat true horizon lines and further refinement becomesnecessary. To prepare for the refinement of the attitude estimate, onlythe edge pixels originating from the horizon in the image are to beextracted. In one example, this is performed geometrically. Edge pixelsclose to the estimated horizon from the Hough voting are then extracted.For a calibrated camera and knowledge of the camera height, the ellipseon the image plane corresponding to the estimated horizon from the Houghvoting will always be slightly smaller than the true perceived horizonon the image due to the topography on top of the ideal spherical earth.Thus, most of the true horizon edge pixels will be on or outside theestimated horizon. Due to quantization effects in the edge detector,some true horizon edge pixels may be ½ pixel inside the estimatedhorizon. If the shift of the horizon due to topography is less than 1pixel, it is sufficient to project the estimated horizon on the imageplane and extract all edge pixels that are within a 3×3 matrix from thehorizon pixels on the image plane.

For high resolution images, and when the ground elevation in the sceneis large, the shift of the horizon due to topography may be larger than1 pixel. The shift from the ideal horizon is in one example in the order4-5 pixels. To extract all true horizon edge pixels, the angular shifton the unit sphere generated by the highest elevation in the surroundingarea compared to sea level is computed. The upper angular limit isherein denoted β_(lim). This means that all true horizon pixels on theimage will be projected onto the unit sphere in a band above theestimated horizon as given by the probabilistic Hough voting. This willbe illustrated in FIG. 9.

The fifth step, computing the geometric horizon comprises in one exampleat least some of the following. For all extracted edge pixels in thefourth step, project the edge pixels onto the unit sphere. Denote thisset of edge pixels P_(s). Rotate the point set with the transpose of theestimated rotation matrix for the vehicle from the Hough voting anddenote the rotated point set P_(r).

Pr=R _(est) ^(T) Ps

The rotated point set on the unit sphere will ideally be the horizonedge pixels rotated to a coordinate system aligned with the worldcoordinate system. The azimuth angles α_(i) for all rotated edge pixelsin P_(r) are computed as

α_(i) =a tan(Pr _(y,i) /Pr _(x,i)).

Based on the assumed 3D position for the vehicle, elevation profiles areextracted from the digital elevation model in all directions α_(i). Foreach direction α_(i), the maximum incidence angle ξ_(max,i) on thecamera is determined based on the elevation profile and the look-uptable. This will generate the geometrically computed horizon

P _(geom,i)=[cos α_(i) sin α_(i) cos(π−ξ_(max,i))]^(T)

The sixth step, refining the pitch and/or roll angle estimates comprisesin one example at least some of the following. The pitch and/or rollangles are refined by minimizing the distance between the point setP_(r), the rotated back projected horizon edge pixels, and the point setP_(geom), the geometrically computed horizon.

In FIG. 9, a band on the unit sphere with conceivable true horizon edgepixels is illustrated. In FIG. 9, a line denoted I_(R) in the figure isthe estimated horizon. A line denoted I_(B) is generated by points thatmake an angle β_(lim) with the horizon points. The line I_(R) is definedby the following equations.

I _(s) ={x _(s) |x _(s) ² +y _(s) ² +z _(s) ²=1Λz _(s)=cos(π−ξ(h _(c)))}

I _(R) ={x _(R) |x _(R) =R(θ_(est),φ_(est))x _(S) , x _(s) εI _(s)}

The line I_(s) is the projection of the horizon onto the unit sphere fora vertically oriented camera at height h_(c) where ξ(h_(c)) is thecorresponding incidence angle on the camera. The line I_(s) is thenrotated with the estimated pitch and roll angles.

The line I_(B) is defined by the following equations.

I _(sh) ={x _(sh) |x _(sh) ² +y _(sh) ² +z _(sh) ²=1Λz _(sh)=cos(π−ξ(h_(c))−ξ_(lim))}

I _(B) ={x _(B) |x _(B) =R(θ_(est),φ_(est))x _(sh) , x _(sh) εI _(sx)}

Compared to I_(s), the line I_(sh) is shifted an angle β_(lim) upwardson the unit sphere. In one example the band between the lines I_(R) andI_(B) is projected onto the image plane to create a mask for potentialhorizon edge pixels. From the Canny edge image, only the edge pixelswithin the mask are extracted for the subsequent attitude refiningprocess.

In FIG. 10, a system 900 for estimating information related to a vehiclepitch and/or roll angle is illustrated. The system 900 comprises aprocessing unit 902 having a first pitch and/or roll angle relatedestimate obtaining element 908 arranged to obtain a first estimate ofthe information related to the pitch and/or roll angle. The processingunit 902 further comprises an improved pitch and/or roll angle relatedestimate obtaining element 909 arranged to determine an improvedestimate of the information related to the pitch and/or roll angle. Thesystem comprises further at least one camera 901 mounted on the airbornevehicle. The at least one camera is arranged to capture an image of anarea covering at least a part of the horizon. The camera is in oneexample an omni-directional camera. The system comprises further adatabase 903 arranged to store a digital elevation or surface model. Theimproved estimate pitch and/or roll angle related obtaining element 909is arranged to determine the improved estimate of the informationrelated to the pitch and/or roll angle based on the first estimate ofthe pitch and/or roll angle related information, and the digitalelevation model.

The first pitch and/or roll angle related estimate obtaining element 908is in one example arranged to obtain the first estimate of the pitchand/or roll angle related information based on a first horizon estimate.In one example, the first horizon estimate is based on an assumptionthat earth is spherical with no topography. The first estimate is in oneexample obtained from the image.

The improved pitch and/or roll angle related estimate obtaining element909 is in one example arranged to obtain a second horizon estimate basedon the digital elevation model, and to determine the improved estimateof the pitch and/or roll angle related information based on the obtainedsecond horizon estimate. In detail, the improved pitch and/or roll anglerelated estimate obtaining element 909 is in one example arranged todetermine the second horizon estimate by: extracting the elevationprofile in all angular directions α_(i) around the vehicle from thedigital elevation model based on a vehicle XY position, determining thelargest incidence angle on the camera, that is generated by all landobjects at heights and distances given by the extracted elevationprofile, and determining for each angular direction α_(i) the maximum ofall incidence angles along the elevation profile to be the geometricalhorizon point where the elevation profile is valid.

In one example, the maximum of all incidence angles along the elevationprofile is determined for each angular direction α_(i) is determinedbased on an extensive search. The system comprises in one example alook-up table 910. The look-up table comprises information related toincidence angles on the camera for substantially all objects along theelevation profile. In one example, the look-up table 910 comprises inputparameters related to camera height, object height and/or distance fromthe camera; and an output parameter related to incidence angle on thecamera.

In one example, the first pitch/roll angle related estimate obtainingelement 908 is arranged to obtain a first horizon estimate based on thecaptured image. The improved pitch/roll angle related estimate obtainingelement 909 is arranged to adjust the first estimate of the pitch and/orroll related information based on a relation between the first horizonestimate and the second horizon estimate so as to determine the improvedestimate.

In one example, the improved pitch and/or roll angle related estimateobtaining element 909 is arranged to determine a plurality of candidatesecond horizon estimates. In one example, each candidate second horizonestimate is related to a candidate camera and/or vehicle threedimensional position. The determination of the improved estimate of theinformation related to the pitch and/or roll angle then involvesselecting that candidate second horizon estimate which provides a bestfit between the first horizon estimate and the second horizon estimate.

The first pitch and/or roll angle related estimate obtaining element 908is arranged to back project the determined edge pixels onto the unitsphere, and determine the first estimate of the pitch and/or rollrelated information based on the back projected edge pixels.

The first pitch and/or roll angle related estimate obtaining element 908is in one example arranged to perform probabilistic voting such asprobabilistic Hough voting for all edge pixels so as to determine thefirst estimate of the pitch and/or roll related information based on theback projected edge pixels.

The first pitch and/or roll angle related estimate obtaining element 908is in one example further arranged to determine the first horizonestimate based on the back projected edge pixels.

The first pitch and/or roll angle related estimate obtaining element 908is further arranged to extract those edge pixel which are within adetermined range from the first horizon estimate and to project theextracted edge pixels onto the unit sphere so as to provide an updatedfirst horizon estimate. The improved pitch and/or roll angle relatedestimate obtaining element 909 is then arranged to adjust the firstestimate of the pitch and/or roll angle related information based on arelation between the updated first horizon estimate and the secondhorizon estimate so as to determine the improved estimate of the pitchand/or roll angle related information.

1-24. (canceled)
 25. A method (200) for estimating information relatedto at least one of a vehicle pitch or roll angle, the method comprisingthe steps of: obtaining (220; 520) a first estimate of the informationrelated to at least one of the pitch or roll angle; capturing (210) animage of an area covering at least a part of the horizon using a cameramounted on the airborne vehicle; and determining (230, 240) an improvedestimate of the information related to at least one of the pitch or rollangle based on the first estimate of the information related to at leastone of the pitch or roll angle, and a digital elevation model.
 26. Amethod according to claim 25, wherein the camera is an omni-directionalcamera.
 27. A method according to claim 25, wherein the first estimateof the information related to the pitch and/roll angle is obtained (210;510) based on a first horizon estimate, wherein the earth is assumed tobe spherical with no topography.
 28. A method according to claim 25,further comprising a step of obtaining (230) at least one second horizonestimate based on the digital elevation model, wherein the determinationof the improved estimate of the information related to the pitch and/orroll angle estimate is based on the obtained at least one second horizonestimate and preferably the first horizon estimate.
 29. A methodaccording to claim 28, wherein the step of obtaining the at least onesecond horizon estimate comprises the following steps: obtaining (331)the respective elevation profile for a plurality of angular directionsα_(i) around the vehicle from the digital elevation model based on avehicle and/or camera three dimensional position; determining (332) thelargest incidence angle on the camera, that is generated by all landobjects at heights and distances given by the extracted elevationprofile; and determining (333) for each angular direction αi along therespective elevation profile the maximum of all incidence angles to bethe geometrical effective horizon point.
 30. A method according to claim28, wherein: a plurality of candidate second horizon estimates aredetermined each related to at least one of a candidate camera or vehiclethree dimensional position; and the determination of an improvedestimate of the information related to the pitch and/or roll angleinvolves selecting that candidate second horizon estimate which providesa best fit between the first horizon estimate and the second horizonestimate.
 31. A method according to claim 30, wherein the at least oneof the camera or vehicle three dimensional position related to theselected candidate second horizon estimate is determined to be the truecamera and/or vehicle three dimensional position.
 32. A method accordingto claim 25, wherein the step of obtaining (220) the first estimate ofthe pitch and/or roll angle related information comprises obtaining thefirst horizon estimate based on the captured image and wherein the stepof determining (240) an improved estimate of the pitch and/or roll anglerelated information comprises adjusting the first estimate of the pitchand/or roll angle related information based on a relation between thefirst horizon estimate and the second horizon estimate.
 33. A methodaccording to claim 25, wherein the determination (510) of the firstestimate of the pitch and/or roll angle related information comprises:back projecting (522) determined horizon edge pixels in the image ontothe unit sphere; and determining (523) the first estimate of the pitchand/or roll related information based on the back projected edge pixels.34. A method according to claim 33, wherein the determination (523) ofthe first estimate of the pitch and/or roll related information based onthe back projected edge pixels comprises probabilistic voting such asprobabilistic Hough voting for all edge pixels.
 35. A method accordingto claim 33, further comprising determining the first horizon estimatebased on the back projected edge pixels.
 36. A method according to claim35, further comprising: extracting those edge pixel which are within adetermined range from the first horizon estimate and projecting theextracted edge pixels onto the unit sphere so as to provide an updatedfirst horizon estimate, wherein the step of determining an improvedestimate of the pitch and/or roll related information comprisesadjusting the first estimate of the pitch and/or roll relatedinformation based on a relation between the updated first horizonestimate and the second horizon estimate.
 37. A system (900) forestimating information related to a vehicle pitch and/or roll angle, thesystem comprising: a processing unit (102; 902) having a first pitchand/or roll angle related estimate obtaining element (908) configured toobtain a first estimate of the information related to the pitch and/orroll angle; at least one camera (101; 901) mounted on the airbornevehicle, said camera being configured to capture an image of an areacovering at least a part of the horizon; and a database (103; 903)configured to store a digital elevation model; wherein the processingunit (102; 902) comprises an improved pitch and/or roll angle relatedestimate obtaining element (909) configured to determine an improvedestimate of the information related to the pitch and/or roll angle basedon the first estimate of the pitch and/or roll angle relatedinformation, and the digital elevation model.
 38. A system according toclaim 37, wherein the camera (101; 901) is an omni-directional camera.39. A system according to claim 37, wherein the first pitch and/or rollangle related estimate obtaining element (908) is configured to obtainthe first estimate of the pitch and/or roll angle related informationbased on a first horizon estimate, wherein the earth is assumed to bespherical with no topography.
 40. A system according to claim 37,wherein the improved pitch and/or roll angle related estimate obtainingelement (909) is configured to obtain at least a second horizon estimatebased on the digital elevation model, and to determine the improvedestimate of the pitch and/or roll angle related information based on theobtained at least one second horizon estimate.
 41. A system according toclaim 40, wherein the improved pitch and/or roll angle related estimateobtaining element (909) is configured to obtain the camera and/orvehicle three dimensional position related to the selected candidatesecond horizon estimate as the true camera and/or vehicle threedimensional position.
 42. A system according to claim 40, wherein theimproved pitch and/or roll angle related estimate obtaining element(909) is configured to determine the at least one second horizonestimate by: extracting the elevation profile in all angular directionsαi around the vehicle from the digital elevation model based on avehicle three dimensional position, determining the largest incidenceangle on the camera, that is generated by all land objects at heightsand distances given by the extracted elevation profile, and determiningfor each angular direction αi the maximum of all incidence angles alongthe elevation profile to be the geometrical horizon point where theelevation profile is valid.
 43. A system according to claim 40, wherein:the first pitch and/or roll angle related estimate obtaining element(908) is configured to obtain a first horizon estimate based on thecaptured image; and the improved estimate obtaining element isconfigured to adjust the first estimate of the pitch and/or roll anglerelated information based on a relation between the first horizonestimate and the at least one second horizon estimate so as to determinethe improved estimate of the pitch and/or roll angle relatedinformation.
 44. A system according to claim 37, wherein the first pitchand/or roll angle related estimate obtaining element (908) is configuredto, back project determined edge pixels onto the unit sphere, anddetermine the first estimate of the pitch and/or roll angle relatedinformation based on the back projected edge pixels.
 45. A systemaccording to claim 44, wherein the first pitch and/or roll angle relatedestimate obtaining element (908) is configured to perform probabilisticvoting such as probabilistic Hough voting for all edge pixels so as todetermine the first estimate of the pitch and/or roll angle relatedinformation based on the back projected edge pixels.
 46. A methodaccording to claim 44, wherein the first pitch and/or roll angle relatedestimate obtaining element (908) further is configured to determine thefirst horizon estimate based on the back projected edge pixels.
 47. Asystem according to claim 46, wherein: the first pitch and/or roll anglerelated estimate obtaining element (908) further is configured toextract those edge pixel which are within a determined range from thefirst horizon estimate and to project the extracted edge pixels onto theunit sphere so as to provide an updated first horizon estimate; and theimproved estimate obtaining element (909) is configured to adjust thefirst estimate based on a relation between the updated first horizonestimate and the second horizon estimate so as to determine the improvedestimate.
 48. A computer program product comprising at least onenon-transitory computer-readable storage medium having computer-readableprogram code portions stored therein, the computer-readable program codeportions comprising at least one executable portion configured forexecuting the steps of the method for estimating information related toa vehicle pitch and/or roll angle according to claim 25.